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ABSTRACT 

This paper examines the concepts of program 
monitoring and program evaluation in the literature, and offers 
working definitions based on two dimensions of measurement: focus 
(what questions are addressed) and timing (how often the measures are 
taken). Focus can be on inputs to the program or outcomes from it; 
timing can be one-shot or continuous. Continuous measures are here 
considered to be monitoring — n input monitoring" if the measures are 
on inputs, "impact monitoring" if they are on outcomes. One-shot 
measures are considered to be evaluation — "process evaluation" if it 
measures inputs, "impact evaluation" if it measures outcome. The 
paper further defines program monitoring as part of the evaluation 
process, and notes that program evaluators must work with project 
managers in making informed decisions, avoiding information overload, 
and implementing and monitoring effective programs. The different 
definitions of program monitoring by Rossi, Freeman, and Wright by 
the Evaluation Research Society, by the World Bank; by J. Wholey, and 
by the International Statistical Program Center are considered and 
compared. (JGL) 
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AN EXAMINATION OF THE CONCEPT AND ROLE OF PROGRAM 
MONITORING AND EVALUATION 

Liese Sherwood- Fabre 
U.S. Bureau of the Census 

ABSTRACT 

The distinction between program monitoring and evaluation is not 
always a clear one. While the evaluation profession has included monitor- 
ing as one type of evaluation, the definitions of each vary greatly within 
the profession. This paper examines the concept of monitoring and 
evaluation that appears in the literature and offers a working definition 
based on the focus and frequency of measurement- This definition allows 
further examination into the requirements for the evaluator when develop- 
ing a program monitoring system and what constitutes a well -designed 
monitoring system. The relationship of program monitoring as part of an 
integrated program monitoring and evaluation system is also discussed. 
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AN EXAMINATION OF THE CONCEPT AND ROLE OF PROGRAM 
MONITORING AND EVALUATION 

This paper represents frustrations and insights I encoun- 
tered during my preparation for a workshop on monitoring and 
evaluation given by the International Statistical Programs Center 
(ISPC), Census Bureau. The frustrations occurred when I began to 
look for reading materials related to program monitoring and 
tried to develop a good, comprehensive definition of monitoring 
from what I found. The insights reflect my discussion with 
colleagues, my reading, and my attempts to synthesize what I 
1 earned • 

The frustrations began when I discovered the scarcity of 
texts on program monitoring. The Eval uati on ' Research Society 
Standards for Program Evaluation (ERS Council, 1982) describe 
program monitoring as "the least acknowledged but probably the 
most practiced category of evaluation . . . ."--which may explain 
why so few writings devoted to program monitoring exist. While 
the most recognized texts may spend a chapter or two discussing 
program monitoring (see, for example, Rossi et al , 1979), I found 
no well-known texts that paid it any extensive attention. Even 
government publications titled "program monitoring" treated the 
topic as it related to fiscal accounting and compliance with 
government regulations. The World Bank has published a book on 
monitoring and evaluation (Casley and Lury, 1982), but it is one 
of the few which even mentions the subject in its title. In this 
paper, I would like to explore the definitions and concepts of 
monitoring which I encountered and how they relate to evaluation 
and the job of the evaluator. In the process, I will bring as 



ERIC 



4 



2 

many of the definitions together as possible and set the context 
for the papers which follow. 

The Increase in the Interest in Program Monitoring ~" 

Prior to my arrival at ISPC, my job had not involved working 
with or developing monitoring systems. I found, however, that 
within the international setting, great efforts exist in develop- 
ing and using monitoring and information systems, and two major 
rorces have encouraged their use. The first involves the 
disenchantment with long-term impact evaluations so popular in 
the 1960s and '70s. While decision makers still recognize these 
as having a place within the field of evaluation, the late 1970s 
and '80s saw a search for techniques which respond more effec- 
tively to managers' and other decision makers' needs for timely, 
usable information. Recently, organizations such as the World 
Bank and the Agency for International Development (AID) have 
shown increased attention to shorter-term evaluations and program 
monitoring. Because of our close association with these agen- 
cies, our own work has shifted as well. 

The introduction of small, low-cost computer systems also 
encouraged the rising interest in monitoring systems. These 
systems complete data processing and analysis much more quickly 
and in greater quantity than ever before. Even developing 
countries can now afford more computer systems with which to 
store and analyze data. This advance has resulted in a tremen- 
dous increase in the amount of information stored which can be 
processed and handed on to managers. For managers, this can 
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often mean that such systems involve the collection of a great 
deal of useless information which makes their job more dif- 
ficult—instead of easier (Patton, 1982: 227). Thus, com- 
puterized monitoring systems only contribute to program manage- 
ment if properly planned and developed. 

The professional evaluator who wishes to continue to be a 
positive, vital influence must become familiar with the role of 
program monitoring within evaluation and understand what it means 
to develop useful and workable systems for program management. 
The evaluator should view program monitoring and information 
systems as one component of an overall, comprehensive evaluation 
plan. As such, it is important to understand how this portion 
operates within the entire plan. 

Some Definitions of Program Monitoring 

Once I found some literature related to program monitoring, 
it became apparent that no two authors agree on what monitoring 
is or what should be monitored. The magnitude of these dif- 
ferences appears as one begins to review what each implies (these 
are summarized in table 1). 

To begin with one of the classic evaluation texts, Rossi, 
Freeman, and Wright (1979: 16) defined monitoring as the "assess- 
ment of whether or not a program is (1) operating in conformity 
to its design, and (2) res ;hing its specified target population." 
From their discussion, I was unable to determine when or how 
often such an assessment should be completed, and inferred that 
monitoring could involve a one-time study done to determine 
whether it is in compliance with its original plan. 



Table 1 

A Comparison of Different Definitions of 
Monitoring 



Def i ni ti on 



Focus 



Timing 



Concept of 
Project 



Rossi , Freeman 
and Wright 



Compl i ance 
with original 
design and 
target pop- 
ul ation. 



Possibly one 
time 



Not Mentioned 



ERS Standards 



Program com- 
pl i ance, 
tracking 
servi ces , 
and counting 
cl ients . 



Conti nuous 



On-goi ng 



World Bank 



Project inputs 
and initial 
effects . 



Continuous 



With definite 
begi nni ng 



Who! ey 



Program 
effects and 
process . 



Conti nuous 



Not Mentioned 



ISPC 



Project inputs 
and outputs. 



Continuous With definite 

beginning 
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The ERS standards (ERS Council, 1982) also include program 
compliance with its definition of monitoring, but goes on to 
include "relatively straightforward tracking of services de- 
• livered and counting of clients." This description emphasizes 
"repeated measurements of program activities related to policy and 
to services as well as clients, -t also implies a concept of 
monitoring as a tool for program management of an on-going 
project. 

The World Bank related monitoring to two other forms of 
evaluation (Casley and Lury, 1982: 4). For these authors, 
"monitoring assesses whether project i nputs are being delivered, 
are being used as intended, and are having the initial effects as 
planned .... Eva! uation assesses the overall project effects , 
both intentional and unintentional, and the impact .... [A]n 
on-going evaluation . . . examine[s] the assumptions and premises 
on which the project design was based" [italics in the original]. 
Here, monitoring goes beyond just program inputs to include some 
effects as well - -al though evaluation still involves the majority 
of the possible program results — and begins at project inception. 

Wholey (1983: 154) refers to tracking program effects as 
"outcome monitoring." This type of monitoring concerns program 
performance, although he notes that process monitoring is also 
important. For outcome monitoring, however, the evaluator works 
on developing agreed-on program objectives and performance 
measures which managers can use to assist their decision making. 
This emphasis on the regular study of immediate outcomes reflects 
Wholey's attention to "results-oriented" managers. 
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Ar, part of its own efforts in the field of evaluation, ISPC 
adapted and refined a definition of monitoring related to 
program development which drew upon involvement in international 
discussions related to monitoring and evaluation. In the model 
used by ISPC, four development components exist: program inputs 
(what is required for program implementation: the resources, 
money, staff, etc.), outputs (the immediate physical results from 
the inputs: clients trained, miles of road completed, wells dug, 
etc.), effects (the changes which result from the interactions 
between the outputs and the target population: changes in client 
behavior, the number of children immunized, an increased use of 
potable water, etc.), and impact (the far-reaching goal or long- 
term benefits: an increase in client income, a decrease in 
mortality, etc.)* Monitoring involves periodic review of program 
inputs and outputs while evaluation focuses on measuring its 
impact, and on-going evaluation on periodic measurement of the 
assumed relationship between outputs and effects. Although 
similar to the World Bank's definition, ISPC emphasizes following 
the program's beginning and implementation when developing a 
monitoring strategy, leaving any effects as part of on-going 
eval uati on . 

These five definitions show what /ariety exists. Although 
repeated data collection occurs in most, it does not explicitly 
appear in Rossi et al's definition. In addition, their defini- 
tion does not include the program effects or outcomes that some 
include. The matter is further complicated when one begins to 
think of the many types of data collection possible. For 
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example, what does one call a study which periodically collects 
data related to a program's impact--such as observing changes in 
the morbidity rates in an area? What about the case where, as 
part of a final evaluation, one collects data on the target 
"■population and compares it to the program's clientele?* Before 
we can intelligently discuss the role of monitoring within 
evaluation, or how to develop a good monitoring system, we must 
have a working definition of monitoring that allows for common 
understandi ng . 

Towards a Working Definition of Monitoring 

This section evolved from a discussion I had concerning the 
very problems I experienced above. A colleague noted that 
perhaps the problem with trying to clearly distinguish between 
monitoring and impact evaluation occurs because they are really 
on a continuum with one end being the impact evaluation and the 
other being the very regular monitoring of program inputs. 

Considering all forms of evaluation on a continuum proved 
very helpful, but as I began to consider the characteristics of 
evaluation, I realized it required more than one dimension. As 
one can see from the discussion in the previous section, two 
major characteristics appear: its focus (or the question it 
answers), and its timing (how often it is done). If these two 
dimensions are crossed, we have a basis for defining both 
monitoring and impact evaluations and the range between (see 
f i gure 1) ♦ 

The first dimension, focus, stretches from the most basic 
program input information which most resembles an accounting 
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Figure 1 

A Two Dimensional Concept of the Different 
Types of Evaluation 
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system to the broadest impact questions which seek to link 
observed changes with the program under study. The second 
dimension, timing, involves moving from frequent, repetitive data 
collection or measurement to one-time only data collection/ 
measurement . 

The results of crossing these two dimensions provide a 
better understanding of how the different definitions described 
above array themselves. The upper left-hand quadrant corresponds 
to the definition offered by Rossi et al . These are studies 
which co'llect data one time related to the program's inputs (and 
possibly outputs). The upper right-hand quadrant is the classic 
evaluation design where infrequent data collection efforts try to 
establish causation between the program and the resulting changes 
(as one moves north along the timing dimension, this would become 
the ex post facto design). Ths lower right-hand quadrant relates 
to Wholey's outcome monitoring. This involves periodic data 
collection related to the changes affected by a program. The 
lower left-hand quadrant is what most would probably agree is 
monitoring: the periodic collection of information related to 
program implementation and activities. For the purposes of this 
panel, we will use "monitoring" in its most generic sense to 
include both the 1 ower ,1 eft-hand and right-hand quadrants and 
"impact evaluation" for the upper right-hand quadrant. 

Thus, data collection for monitoring purposes would address 
such program management questions as "Are the materials needed 
for program implementation arriving as scheduled and when 
needed?", "Is the program implementation progressing as sche- 
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duled?" and "Are there observable changes occurring?" Impact 
evaluation would consider such issues as "Did the observed 
' changes result from program activities, or were there other 

extraneous conditions which created the changes?" 

The Importance of the Evaluator in Developing a Monitoring System 
For the evaluator, program monitoring represents a chance to 
become involved in everyday program activities and decisions. 
The ERS standards note that this form of evaluation "puts to rest 
the notion that the evaluator necessarily comes in, does the job, 
and then gets out" (ERS Council, 1982) • 

Because it is an evaluation activity, program monitoring 
requires the evaluator to follow the same steps he would in 
designing a long-term impact evaluation. The evaluator must work 
with decision makers in determining what information they need to 
manage a program and how to collect that information. The major 
differences involve the information collected and the schedule of 
results. As we noted above, program monitoring information 
involves following a program's inputs and outputs to ensure that 
it is operating as required and where it is not, to identify 
problem areas. Thus, a monitoring system can note that a problem 
exists and where corrective actions should occur. 

Part of creating a usable system may include helping program 
managers understand how they make decisions, and how to make 
informed ones. Particularly in some of the countries where we 
work, we have found that managers lack training in making 
decisions based on such information. This problem, however, is 
not limited to developing countries. Patton (1982: 229) has 
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found similar problems with social program managers in the U.S. 
as well. Thus, prior to even identifying the questions to be 
answered, the evaluator may have to work witn managers to 
identify the types of decisions they make and the information 
""they need to make them. The next step would be to determine and 
design the data collection and analysis process. 

Once managers see how useful a program monitoring or 
information system can be, the evaluator's role may shift to 
ensure that managers do not become overly enthusiastic and try to 
drastically increase the amount of information collected. The 
storage space and ease of computer outpvit often create a great 
temptation* The evaluator must force managers to decide what 
information is collected based on the criteria of "what dif- 
ference will knowing this information make?" In other words, if 
the information is not related to some specific decision, and if 
the manager will not take some action based on knowing a specific 
piece of information, the monitoring system should not include 
it. This creates a hard role for the evaluator to play, but it 
is important if the information system is to remain useful and 
relevant, and not overload managers with more information than 
they can truly use. 

Implications fcr Evaluators 

In our own work, we have seen a major increase in the use 
and interest in p "ogrem monitoring and information systems. 
While such systems need not be computerizeG, the likelihood is 
that they will be because the computer can make the process and 
analysis stage much easier. This places a burden on the evalu- 
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ator to become sufficiently familiar with computers to know what 
they are capable of doing. I am not suggesting that we become 
data processing experts, but we, as evaluators, should have the 
expertise to help decision makers translate their questions into 
measurable items and to know how to best collect that informa- 
tion • 

This shift fclso implies that we must be sufficiently 
familiar v/ith management activities to be able to discuss 
management and decision making processes intelligently with 
information users. This is particularly true for situations 
where decision making has been done on an ad-hoc or informal 
basi s • 

A final adjustment that evaluators will also have to make is 
how to use program monitoring and information systems as part of 
a comprehensive impact evaluation. The information regularly 
collected as part of the system can become an important resource 
for describing program implementation, process, and even, 
outcomes- In projects which expect the evaluator to design both 
a monitoring and impact evaluation plan, the evaluator should 
also consider how the monitoring information might enhance the 
impact evaluation* 

The purpose of a good program monitoring or information 
system falls easily within the evaluator's domain. Evaluation 
has always had as its main goal providing useful, timely data 
regarding a program to interested parties. Program monitoring 
seeks to provide this information on a regular basis during 
project implementation to help managers effectively manage their 



ERLC 



15 



14 

programs. For the evaluator, this means understanding project 
management from the manager's perspective and knowing where 
program information is most needed and most useful. 
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FOOTNOTES 

Groups such as the World Bank and AID who separate monitor- 
ing from evaluation further complicated my problems. This 
separation, however, appears to result from the disenchant- 
ment I mentioned earlier. "Evaluation," as th.sy use it, 
appears to be the long-term impact evaluation. To avoid 
confusion, then, I will use the term "impact evaluation" fcr 
this type of evaluation. 
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